Shift-curvature, SGD, and generalization

Abstract A longstanding debate surrounds the related hypotheses that low-curvature minima generalize better, and stochastic gradient descent (SGD) discourages curvature. We offer a more complete nuanced view in support of both hypotheses. First, we show curvature harms test performance through two new mechanisms, shift-curvature bias-curvature, addition to known parameter-covariance mechanism. The shift refers difference between train local minima, bias covariance are those parameter distribution. These three curvature-mediated contributions reparametrization-invariant even though itself is not. Although unknown at training time, as well other mechanisms can still be mitigated by minimizing overall Second, derive new, explicit SGD steady-state distribution showing optimizes an effective potential but different from loss, noise mediates trade-off low-loss versus regions this potential. Third, combining our analysis with steady state shows for small noise, dominant mechanisms. Our experiments demonstrate significant impact on further explore relationship